Functions of Finite Markov Chains

Markovian bounds on functions of finite Markov chains

In this paper, we obtain Markovian bounds on a function of a homogeneous discrete time Markov chain. For deriving such bounds, we use well known results on stochastic majorization of Markov chains and the Rogers-Pitman’s lumpability criterion. The proposed method of comparison between functions of Markov chains is not equivalent to generalized coupling method of Markov chains although we obtain...

Finite Markov Chains

This paper provides basic information and theorems about finite Markov chains. The inspiration for this paper came from Professor Laci Babai’s Discrete Mathematics lecture notes from the REU program of 2003. After reading through his chapter on Markov chains, I decided to proceed by answering as many exercises from the notes as possible. Below is what I have finished.

Lectures on finite Markov chains

Self - Learning Control of Finite Markov Chains

This book presents a collection of work on algorithms for learning in Markov decision processes. The problem addressed is very similar in spirit to “the reinforcement learning problem,” which has become a central topic in artificial intelligence (see, e.g., [8]). However, the book reflects an entirely different thread of research, and there has been surprisingly little interaction between this ...

Graph-Theoretic Analysis of Finite Markov Chains

Markov chains arise frequently in the modeling of physical and conceptual processes that evolve over time. For example, the diffusion of liquids across a semiporous membrane, the spread of disease within a population, and the flow of personnel within the ranks of an organization can all be modeled using Markov chains. In each of these cases, the system can be found in any of a finite number of ...